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I.  BACKGROUND 

We  proposed  to  build  a  data  collection  and  software  workbench,  called  the  Workload  Assessment 
Research  workstation  (WAR  Station),  to  facilitate  research  on  mental  workload.  The  WAR  Station  is  to 
include  hardware  and  software  for  collecting  electrophysiological  and  behavioral  data  associated  with  tasks 
used  in  mental  workload  research  and  a  collection  of  software  tools  for  analyzing  these  data  with  the  goal 
of  building  and  testing  practical  mental  workload  indices.  The  indices  would  be  based  on  a  neural- 
network  analytic  paradigm  used  in  our  prior  research  (Gevins,  1980;  Gevins  and  Morgan,  1986,  1988). 

Much  of  this  system  builds  on  hardware  and  software  under  development  in  other  projects  at  SAM 
Technology.  The  major  work  required  of  this  project  includes 

1)  designing,  implementing,  and  testing  an  improved  neural  network  algorithm; 

2)  performing  experiments  to  provide  an  initial  test  of  the  utility  of  our  analytic  approach  for  build¬ 
ing  practical  indices  of  mental  workload; 

3)  building  the  WAR  Station  by  integrating  workload-specific  software  with  software  and  hardware 
components  developed  under  other  projects  at  SAM  Technology;  and 

4)  surveying  Air  Force  researchers  to  determine  desirable  system  features  and  work  out  what  data 
import  capabilities  will  be  needed. 

These  four  items  reflect  an  approved  revision  of  our  Statement  of  Work  which  consisted  of  append¬ 
ing  the  second  item  to  our  work  plan  and  postponing  integration  of  our  system  with  commercial  moving- 
base  flight  simulators  until  Phase  III. 

II.  SUMMARY 

Report  Period,  16  DEC  92  to  31  AUG  94 
Alan  Gevins,  Principal  Investigator 
Harrison  Leong,  Chief  of  Engineering 
Michael  Smith,  Principal  Neuroscientist 
Linda  McEvoy,  Psychophysiologist 
Sue  Whitfield,  Research  Associate 
Georgia  Rush,  Research  Associate 
Paulo  Raffaelli,  Programmer 
Behram  daCosta,  Programmer 

During  this  period  we  collected  a  large  amount  of  data  in  a  working  memory  paradigm  and  have 
performed  much  of  the  initial  data  preprocessing  that  is  prerequisite  to  neural  network  pattern  recognition 
analyses.  We  have  also  designed  a  second  experiment  which  will  allow  us  to  determine  whether 
workload-related  EEG  features  can  be  extracted  from  a  more  demanding  experiment,  one  which  more 
closely  resembles  the  tasks  pilots  are  required  to  perform.  Finally,  several  manuscripts  were  prepared  and 
submitted  for  publication. 
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Le,  J.,  Menon,  V.  and  Gevins,  A.S.  (In  Press,  1994)  Local  Estimate  of  Surface  Laplacian  Derivation  on  a 
Realistically  Shaped  Scalp  Surface.  Electroenceph.  Clin.  Neurophysiol. 

Gevins,  A.S.  (In,  Press,  1994)  Imaging  the  neurocognitive  networks  of  the  human  brain.  In:  Erin  Bigler 
(Ed.),  Handbook  of  Human  Brain  Function:  Neuroimaging,  Plenum  Press. 

Gevins,  A.S.  (In,  Press,  1994)  High  Resolution  EEG  Studies  of  Cognition.  Epilepsy  and  Funtional  Ana¬ 
tomy  of  the  Frontal  Lobe,  Raven  Press. 
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somatosensory  discrimination.  Electroenceph.  Clin.  Neurophysiol.,  Vol.  92  (4),  Elsevier:  Amster¬ 
dam,  pp.  282-290. 

Gevins,  A.S.,  et  al.  (1994)  High  resolution  evoked  potential  technology  for  imaging  neural  networks  of 
cognition.  In:  Thatcher,  R.W.,  et  al.  (Eds.),  Functional  Neuroimaging:  Technical  Foundations. 
Academic  Press,  Inc.  :  Orlando,  pp.  223-231. 

Gevins,  A.S.,  et  al.  (1994)  High  Resolution  EEG:  124-Channel  recording,  spatial  deblurring  and  MRI 
integration  methods.  Elechtroenceph.  Clin.  Neurophysiol.,  90,  Elsevier:  Arnsterdam,  pp.  337-358. 

Gevins,  A.S.,  et  al.  (1994)  Imaging  the  spatiotempral  dynamics  of  cognition  with  high  resolution  evoked 
potential  methods.  Human  Brain  Mapping,  Vol.  1  (2),  John  Wileys:  New  York,  pp.  101-116. 

Gevins,  A.S.  and  Cutillo,  B.A.  (1993)  Spatiotemporal  dynamics  of  component  processes  in  human  work¬ 
ing  memory.  Electroenceph.  Clin.  Neurophysiol.,  Vol.  87,  Elsevier:  Amsterdam,  pp.  128-143. 
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IEEE  Transactions  on  Biomedical  Engineering,  Vol.  40,  No.  6,  pp.  517-528. 

Wikswo  J.P.,  Gevins,  A.S.,  Williamson  S.  J.  (1993)  The  future  of  the  EEG  and  MEG.  Electroenceph 
Clin.  Neurophysiol.,  Vol.  1,  Elsevier:  Amsterdam,  pp.  1-9. 
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IV.  PROGRESS  REPORT 
Working  Memory  Data  Base 

Working  memory,  that  is,  the  process  of  maintaining  information  for  several  seconds  while  it  is 
being  utilized  in  focused  thought,  is  critical  to  performance  in  complex  task  environments.  We  designed 
an  experiment  to  determine  whether  neural  network  based  pattern  recognition  could  use  physiological 
features  to  distinguish  between  3  levels  of  difficulty  on  a  working  memory  task,  whether  these  features 
would  generalize  across  task  versions  that  were  matched  on  stimulus,  response,  and  difficulty  parameters, 
but  that  differed  in  the  type  of  cognitive  processing  required  (spatial  versus  verbal  working  memory),  and 
whether  these  features  were  reliable  enough  to  generalize  across  testing  sessions.  In  the  test  we  imple¬ 
mented,  stimuli  were  presented  in  a  continuous  sequence  and  the  subject  was  required  to  compare  the 
current  stimulus  with  those  earlier  in  the  sequence.  Each  stimulus  was  selected  from  a  set  of  twelve  capi¬ 
tal  letters,  and  displayed  in  one  of  twelve  screen  positions.  Subjects  were  required  to  compare  the  current 
stimulus  to  the  immediately  preceding  stimulus  (low  workload  level)  or  the  the  stimulus  that  occurred  two 
trials  ago  (intermediate  workload  level)  or  the  stimulus  that  occurred  three  trials  ago  (high  workload 
level).  In  the  verbal  processing  task  version,  the  subject  was  required  to  determine  whether  the  current 
stimulus  letter  matched  the  previous  letter.  In  the  spatial  processing  version,  the  task  was  to  determine 
whether  or  not  stimuli  appeared  in  the  same  spatial  location. 

Eight  subjects  participated  in  three  days  of  testing.  The  6  tasks  (three  levels  each  of  the  verbal  and 
spatial  versions)  were  presented  in  blocks  of  20  trials.  Each  subject  performed  8  repetitions  of  each  block 
per  day.  The  blocks  were  presented  in  random  order.  Subjects  made  a  button  press  response  with  the 
right  finger  if  the  current  stimulus  matched  the  target  stimulus  (that  presented  one,  two  or  three  trials  ago), 
or  with  the  left  finger  if  the  current  stimulus  did  not  match  the  target.  After  each  block,  subjects  reported 
their  subjective  impression  of  the  task  by  performing  the  Subjective  Workload  Assessment  Test  (SWAT). 
The  EEG  was  recorded  continuously  from  27  scalp  electrodes  referenced  to  digitally-linked  mastoids,  and 
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the  EOG  was  recorded  from  four  eye  electrodes.  The  passband  was  0.05  to  100  Hz,  and  data  were  sam¬ 
pled  at  256  Hz.  Four  subjects  participated  in  a  call-back  session  approximately  one  month  after  the  initial 
experiment.  Of  these  four  subjects,  three  came  for  a  second  call-back  session  one  month  thereafter.  Each 
of  these  call-back  sessions  comprised  one  day  of  recordings  in  which  subjects  performed  nine  repetitions 
of  each  block.  This  call-back  data  will  be  used  to  determine  the  reliability  of  the  algorithm  to  classify 
data  over  time. 

Analysis  of  the  behavioral  and  subjective  data  indicate  that  the  experimentally-defined  workload  lev¬ 
els  did  indeed  differ  in  difficulty.  Both  reaction  time  and  accuracy  showed  significant  effects  of  workload 
level  with  subjects  showing  a  monotonic  increase  in  reaction  time  and  decrease  in  accuracy  from  the 
lowest  to  the  highest  workload  level.  The  SWAT  measures  indicated  that  subjects  found  the  three  levels  to 
significantly  differ  in  difficulty,  with  the  lowest  level  being  the  easiest  and  the  highest  level  the  hardest. 
No  significant  differences  were  observed  between  the  verbal  and  spatial  versions  of  die  tasks. 

The  physiological  data  have  been  manually  reviewed  for  artifacts.  Eye  blinks  and  eye  movement 
artifacts  have  been  filtered  out  of  the  EEG  data,  and  trials  containing  other  types  of  artifact  have  been 
excluded  from  further  analysis.  Inidal  exploratory  frequency  domain  analyses  have  also  been  performed 
for  several  of  these  subjects  to  determine  whether  reliable  spatiotemporal  features  existed  in  their  EEG 
that  discriminated  workload  levels.  Results  to  date  indicate  that  there  is  sufficient  workload-related  signal 
to  justify  performing  single  trial  pattern  recognition  analyses  on  these  data.  Thus  we  will  first  examine 
whether  such  an  approach  will  be  successful  in  discriminadng  the  three  workload  levels.  If  this  is  suc¬ 
cessful  we  will  then  examine  the  reliability  of  this  index  by  testing  whether  networks  trained  on  the  data 
collected  from  two  days  will  be  able  to  correctly  classify  the  data  collected  on  the  other  day.  Third,  we 
will  examine  the  generality  of  our  workload  index  by  examining  whether  a  network  trained  on  data  from 
the  verbal  condition  can  accurately  classify  workload  levels  in  the  spatial  condition.  Lastly,  we  will  exam¬ 
ine  the  possibility  that  information  content  of  a  task  influences  spatiotemporal  features  of  the  EEG  by 
attempting  to  explicitly  train  a  network  to  discriminate  between  spatial  and  verbal  task  performance. 

Divided  Attention  Data  Base 

Divided  attention,  the  ability  to  simultaneously  allocate  attention  to  multiple  tasks,  is  of  fundamental 
importance  in  piloting  an  aircraft.  We  have  designed  an  experiment  to  assess  subjects’  ability  to  perform 
several  tasks  at  once,  each  of  which  emphasize  a  different  collection  of  mental  resources,  and  have  begun 
to  collect  pilot  data.  EEG,  eye  movement  and  blink  potentials,  and  EKG  will  be  recorded  while  subjects 
perform  three  different  neurocognitive  tasks  simultaneously.  The  subcomponents  of  this  task  exercise 
different  mental  resources;  they  include: 

Working  Memory-maintaining  information  for  several  seconds  while  it  is  being  utilized  in  the  con¬ 
text  of  a  mental  task; 

Dynamic  Visuospatial  Reasoning-rapidly  coordinating  behavior  with  information  about  the  location 
and  speed  of  moving  objects;  and, 

Arithmetic  Reasoning-mentally  solving  simple  mathmetical  equations. 

The  working  memory,  dynamic  visuospatial  reasoning,  and  arithmetic  tasks  are  concurrently 
presented  on  different  regions  of  a  computer  screen.  Each  task  requires  a  speeded  finger-press  response. 
To  make  a  response  to  a  specific  task,  the  subject  must  press  the  left  or  right  button  of  the  left-handed 
mouse  to  move  the  cursor  to  the  desired  task.  The  subject  then  responds  to  the  task  with  a  left  oi  right 
mouse  button  press  of  the  right-handed  mouse.  In  the  working  memory  task,  stimuli  are  presented  con¬ 
tinuously,  and  the  subject  must  compare  the  current  stimulus  with  the  immediately  preceding  stimulus,  and 
make  a  "match"  or  "mismatch"  button  press  response.  In  the  dynamic  visuospatial  reasoning  task  the  sub¬ 
ject  must  make  a  button  press  response  at  the  appropriate  time  so  that  a  "missile”  fired  from  a  stationary 
letter  intercepts  a  target  letter  travelling  vertically  across  the  screen.  In  the  arithmetic  reasoning  task,  the 
subject  must  mentally  solve  a  simple  mathematical  equation  and  indicate  with  the  appropriate  button  press, 
whether  the  outcome  is  greater  or  less  than  5.  Each  task  has  a  limited  time  for  response,  so  subjects 
must  move  quickly  among  the  tasks  to  perform  well  in  the  experiment.  Each  task  has  a  control  level:  in 
the  visuospatial  task,  any  button  press  during  the  task  will  successfully  intercept  the  target;  in  the  arith¬ 
metic  and  working  memory  tasks,  the  words  "left"  and  "right"  will  appear,  requiring  a  left  or  right  button 
press  response.  The  control  tasks  thus  have  similar  visual  and  motor  requirements  as  the  active  tasks,  but 
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do  not  require  the  subjects  to  solve  the  tasks.  Control  and  active  tasks  will  be  presented  in  different  com¬ 
binations,  ranging  from  all  three  control  tasks  at  once,  to  all  three  active  tasks  at  once.  Intermediate  levels 
will  have  one  or  two  control  tasks  with  two  or  one  active  tasks,  respectively.  Performance  data  and 
SWAT  data  will  be  collected  and  analyzed.  The  physiolgical  data  will  be  analyzed  in  the  same  way  as 
the  working  memory  data.  The  ongoing  EEG  data  will  be  divided  into  short  segments,  and  the  the  fre¬ 
quency  spectra  will  be  calculated.  Spectral  features  most  likely  to  distinguish  between  difficulty  levels 
will  be  subjected  to  neural  network  pattern  recognition  analysis  to  determine  whether  we  can  distinguish 
the  EEG  patterns  associated  with  the  more  demanding  conditions  from  those  associated  with  the  easier 
conditions. 

V.  CONCLUSIONS 

These  experiments  will  provide  the  necessary  database  to  determine  whether  our  neural  network 
approach  can  be  used  to  reliably  classify  different  levels  of  mental  workload.  We  will  begin  the  neural 
network  analysis  on  the  existing,  working  memory  data  set  immediately.  We  will  also  proceed  with  the 
piloting  and  actual  data  collection  for  the  divided  attention  experiment 
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